Technology-Assisted Review in Empirical Medicine: Waterloo Participation in CLEF eHealth 2017

Screening articles for studies to include in systematic reviews is an application of technology-assisted review (“TAR”). In this work, we applied the Baseline Model Implementation (“BMI”) from the TREC Total Recall Track (2015-2016) to the CLEF eHealth 2017 task of screening MEDLINE abstracts to identify articles reporting studies to be considered for inclusion. According to rank-based evaluation measures, this approach identified every article describing a study that should have been included in each of 30 systematic reviews, by examining 461 abstracts, on average, per review—12.6% of the 3, 655 abstracts that would have had to be examined, on average, if instead, a manual approach had been used. While this result indicates TAR’s promise to substantially reduce the time and cost of abstract screening, this promise can be realized only if it can be known with reasonable certainty for each review how many abstracts must be examined before all, or substantially all, articles that should be included have been identified. To this end, we applied our “knee-method” stopping criterion to BMI to determine how many abstracts should be examined for each topic. According to thresholdbased evaluation, the knee method identified every article that should have been included (100% recall), while examining 2, 659 abstracts, on average, per topic—72.8% of the 3, 655 abstracts, that would have required examination, on average, had a manual approach been used instead. While our results suggest that TAR can substantially improve the efficiency of abstract screening without compromising recall, there remains room for improvement both in ranking and stopping criterion, as well as important factors that were not addressed in the CLEF eHealth 2017 framework: the completeness of the universe of abstracts gathered using keyword search, and the accuracy of the human assessments of the collected abstracts.